{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Hackathon #1\n",
    "\n",
    "Written by Eleanor Quint\n",
    "\n",
    "Topics: \n",
    "- TF Graph\n",
    "- Tensors\n",
    "- Sessions\n",
    "- Operations\n",
    "- Placeholders\n",
    "- Variables: initialization and assignment\n",
    "\n",
    "Some material adapted from the TensorFlow documention: https://www.tensorflow.org\n",
    "\n",
    "This is all setup in a IPython notebook so you can run any code you want to experiment with. Feel free to edit any cell, or add some to run your own code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We'll start with our library imports...\n",
    "from __future__ import print_function\n",
    "\n",
    "import tensorflow as tf  # to specify and run computation graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### TF Graph\n",
    "\n",
    "TensorFlow programs typically consist of two sections:\n",
    "1. Building the computation graph\n",
    "2. Running the computation graph\n",
    "\n",
    "A computational graph is a series of TensorFlow operations arranged into a graph of nodes. Each node takes zero or more tensors as inputs and produces a tensor as an output. This is called the tensor-in tensor-out (TITO) model, where we may think of tensors as the edges between node operations.\n",
    "\n",
    "The computation graph is an environment which can be interacted with using Python code, but exists separately from the Python environment entirely. We'll cover the basics of how this interaction works today.\n",
    "\n",
    "#### Tensor\n",
    "\n",
    "The basic unit of data in TensorFlow is the [Tensor](https://www.tensorflow.org/api_docs/python/tf/Tensor). A tensor consists of a set of primitive values (think `float` or `int`) shaped into an n-dimensional array. We say that a tensor's _rank_ is its number of dimensions. Here are some examples of tensors:\n",
    "```\n",
    "its_complicated = tf.Variable(12.3 - 4.85j, tf.complex64)   # a rank 0 tensor; a scalar with shape []\n",
    "first_primes = tf.Variable([2, 3, 5, 7, 11], tf.int32)      # a rank 1 tensor; a vector with shape [5]\n",
    "myxor = tf.Variable([[False, True],[True, False]], tf.bool) # a rank 2 tensor; a matrix with shape [2, 2]\n",
    "my_image = tf.zeros([10, 299, 299, 3])                      # a rank 4 tensor with shape [10, 299, 299, 3]\n",
    "```\n",
    "The rank of a tensor can be retrieved with [tf.rank](https://www.tensorflow.org/api_docs/python/tf/rank). There are two ways of accessing the shape of a `tf.Tensor`. The first is used during graph construction by reading the `shape` property of a tensor (e.g., `node1.shape`), which returns a [TensorShape](https://www.tensorflow.org/api_docs/python/tf/TensorShape) object. The second, at runtime, can be done by calling the [tf.shape](https://www.tensorflow.org/api_docs/python/tf/shape) operation. Tensors can be reshaped with [tf.reshape](https://www.tensorflow.org/api_docs/python/tf/reshape).\n",
    "\n",
    "\n",
    "One simple type of graph node is a [constant](https://www.tensorflow.org/api_docs/python/tf/constant):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "node1 = tf.constant(3.0, dtype=tf.float32)\n",
    "node2 = tf.constant(4.0, name='four_const') # also tf.float32 implicitly\n",
    "print(node1, node2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that printing the nodes does not output the values `3.0` and `4.0` as you might expect. This is all information available statically about the `Tensor`s, before initializing the graph or running anything. We can see that each `Tensor` has a name, a shape, and a data type.\n",
    "\n",
    "#### TF Session\n",
    "\n",
    "Now, we have to create a TensorFlow [Session](https://www.tensorflow.org/api_docs/python/tf/Session) which encapsulates the TF environment in which to run our graph. Then, we can use [Session.run](https://www.tensorflow.org/api_docs/python/tf/Session#run) or [tf.Tensor.eval](https://www.tensorflow.org/api_docs/python/tf/Tensor#eval) to get the value of a Tensor out of the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "sess = tf.Session()\n",
    "print(sess.run([node1, node2])) # This prints what we expect\n",
    "print(node1.eval(session=sess)) # Another way of getting the value \n",
    "with sess.as_default():\n",
    "    print(node2.eval())         # This also works... "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each `Tensor` is uniquely identified by its name in the namespace of a `Graph`. No more than one `Tensor` can be declared with a particular name and a `Tensor` can be uniquely identified and retrieved by its name. If you declare the same name twice, the name will be uniquified automatically to avoid collision. The python object returned from a function call is just a handle (like a pointer) to the corresponding object in the TensorFlow graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "node2_clone = tf.constant(4.0, name='four_const')\n",
    "with sess.as_default():\n",
    "    print(node2_clone.name) # notice that this has been uniquified"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Operations\n",
    "\n",
    "Next, we'll add an [Operation](https://www.tensorflow.org/api_docs/python/tf/Operation) to the graph by adding together the constants we declared earlier. The operation is in the TF graph, so initially it return a `Tensor` object and we'll have to use [Session.run](https://www.tensorflow.org/api_docs/python/tf/Session#run) on the output `Tensor` to run the operation and get the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# tf.add sums the two tensors provided\n",
    "node3 = tf.add(node1, node2)\n",
    "print(\"node3:\", node3)\n",
    "print(\"sess.run(node3):\", sess.run(node3))\n",
    "\n",
    "# list all operations\n",
    "tf.get_default_graph().get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that when we print `node3`, it's a `Tensor` whose name ends with `:0`. The name of a `Tensor` usually ends with `:i` where `i` is an integer. This indicates that it's the `i`th output of the `Operation` of the same name. Internally, everything including constants and Variables are operations with output Tensors.\n",
    "\n",
    "#### Placeholders\n",
    "\n",
    "A graph can be setup to accept external inputs using [tf.placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = tf.placeholder(tf.float32)\n",
    "b = tf.placeholder(tf.float32)\n",
    "adder_node = a + b  # '+' provides a shortcut for tf.add(a, b)\n",
    "print(\"a:\", a)\n",
    "print(\"b:\", b)\n",
    "print(\"adder_node:\", adder_node)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many python operators are overloaded by TensorFlow to be their pointwise equivalents using numpy broadcasting (you can see the details [here](https://www.tensorflow.org/api_docs/python/tf/Tensor#methods), where everything with `__op__` form should be an overload). Much of the TensorFlow API closely resembles that of Numpy, including built-in features like [broadcasting]( https://docs.scipy.org/doc/numpy-1.13.0/user/basics.broadcasting.html). This is important to understand in order to effectively use TensorFlow!\n",
    "\n",
    "Now, the graph looks like this:\n",
    "![img_2](https://www.tensorflow.org/images/getting_started_adder.png)\n",
    "\n",
    "In order to use operations that rely directly or indirectly on placeholders we must provide a `feed_dict` to the `Session.run` method. This is a dictionary with `placeholder` as keys and the value that they should use as the corresponding values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(sess.run(adder_node, {a:3, b: 4.5}))\n",
    "print(sess.run(adder_node, feed_dict={a: [1, 3], b: [2, 4]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that, in the second `print` we feed lists for the variables. Python lists and `numpy` arrays can both be converted into Tensors.\n",
    "\n",
    "We can make the computational graph more complex by adding another operation. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "add_and_triple = adder_node * 3.\n",
    "print(sess.run(add_and_triple, {a: 3, b: 4.5}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph now looks like this:\n",
    "![img_3](https://www.tensorflow.org/images/getting_started_triple.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hackathon 1 Exercise 1\n",
    "\n",
    "Write code to evaluate the function `f(x, y) = 7xy^2*cos(3x) + sqrt(5)*xy + exp(2y)` in the cell below using placeholders for `x` and `y`, and evaluate the function in a session. The output should be the result of the function for your choice of values of `x` and `y`. To get points for this hackathon, make sure you fill this out, save and download the notebook, and submit it on Handin. (If you're having trouble, look at the [TF documentation](https://www.tensorflow.org/api_docs/python/tf))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variables and initialization\n",
    "\n",
    "Let's imagine now that we have a dataset of `(input, output)` pairs to which we want to fit a linear function.\n",
    "\n",
    "To make the model store the parameters of the linear function, we need to be able to modify the graph to get new outputs with the same input. [Variables](https://www.tensorflow.org/api_docs/python/tf/Variable) allow us to add trainable parameters to a graph. They are constructed with a type and initial value and must be explicitly initialized.\n",
    "\n",
    "High-level `Variable` how-to: https://www.tensorflow.org/programmers_guide/variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W = tf.Variable([.3], dtype=tf.float32)  # slope\n",
    "b = tf.Variable([-.2], dtype=tf.float32) # y-intercept\n",
    "x = tf.placeholder(tf.float32)           # input placeholder\n",
    "linear_model = W*x + b                   # define output\n",
    "print(\"W:\", W)\n",
    "print(\"b:\", b)\n",
    "print(\"linear_model:\", linear_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Constants are initialized when you call `tf.constant`, and their value can never change.\n",
    "By contrast, variables are not initialized when you call `tf.Variable`.\n",
    "To initialize all the variables in a TensorFlow program, we can explicitly call a special operation, [tf.global_variables_initializer](https://www.tensorflow.org/api_docs/python/tf/global_variables_initializer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)\n",
    "# Now we can do things with the values of the Variables we defined\n",
    "print(sess.run([W, b]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since `x` is a placeholder, we can evaluate `linear_model` for several values of `x` simultaneously by passing a list as input to get a list as output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(sess.run(linear_model, {x: [1, 2, 3, 4]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've created a linear model to fit the data function, but we don't yet know how well it perform. To evaluate the model on training data, we need a `y` placeholder to input the correct output values, and we need to write a loss function.\n",
    "\n",
    "A loss function measures how far the current estimated output is from the correct output and can be used as a signal to update `Variable` values to get a better fitting model. We'll use a standard loss funciton for linear regression, squared loss, which sums the squares of the differences between the model output and the correct output, given an input.\n",
    "\n",
    "`linear_model - y` creates a vector where each element is the corresponding example's error delta. We then call `tf.square` to square that error, and finally we sum all the squared errors to create a single scalar that combines the error of all examples using `tf.reduce_sum`. This gives an aggregate measure of how far the model is from fitting the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = tf.placeholder(tf.float32)\n",
    "squared_deltas = tf.square(linear_model - y)\n",
    "loss = tf.reduce_sum(squared_deltas)\n",
    "print(sess.run(loss, {x: [1, 2, 3, 4], y: [0, -1, -2, -3]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can improve the fit manually by re-assigning the values of `W` and `b` to the perfect values of `-1` and `1` respectively.\n",
    "A variable is initialized to the value provided to `tf.Variable`, but can be changed using operations like [tf.assign](https://www.tensorflow.org/api_docs/python/tf/assign)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fixW = tf.assign(W, [-1.])\n",
    "fixb = tf.assign(b, [1.])\n",
    "sess.run([fixW, fixb])\n",
    "print(sess.run(loss, {x: [1, 2, 3, 4], y: [0, -1, -2, -3]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Typically, rather than providing static values as the second argument to `tf.assign`, a placeholder is created to allow updating repeatedly as needed.\n",
    "\n",
    "We guessed the \"perfect\" values of `W` and `b`, but the whole point of machine learning is to find the correct model parameters automatically. We will show how to accomplish this in the next Hackathon."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "TensorFlow 1.12 (py36)",
   "language": "python",
   "name": "tensorflow-1.12-py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
